1. Field of the Invention
The present invention relates to an image signal processing apparatus which can be applied to an auto-focus camera and an environment recognition apparatus, and is used for distance measurements or defocus amount calculations.
2. Related Background Art
As one of methods of measuring the distance from a measurement apparatus to the object to be measured, a stereoscopic method is known, and is popularly used as an optical measurement means.
FIG. 8 is a view for briefly explaining the stereoscopic method. Referring to FIG. 8, lenses 1 and 2 have a focal length f, and also have optic centers 3 and 4. Optical axes 5 and 6 of the lenses 1 and 2 are assumed to be parallel to each other. Also, assume that each lens is an ideal one, and the influence of aberrations is not taken into consideration. The middle point between the optic centers 3 and 4 of the lenses is set as the origin of a coordinate system, the optical axis direction of each lens is set as an x-axis, and an axis connecting the optic centers 3 and 4 of the lenses is set as a y-axis. Also, an axis perpendicular to the x- and y-axes is set as a z-axis. Note that a plane x=-f corresponds to an imaging surface 7 of the two lenses. Image recording means 8 and 9 record images formed on the imaging surface. Assume that an object 10 to be measured is located at coordinates (x0, y0, z0). Images of the object 10 are formed at positions 11 and 12 by the two lenses. These positions 11 and 12 are those where straight lines connecting the object 10 and the optic centers 3 and 4 cross the imaging surface 7. If positions where the optical axes of the lenses and the imaging surface 7 cross are assumed to be reference positions 13 and 14 of the image recording means 8 and 9, distances p.sub.1 and p.sub.2 between the imaging positions 11 and 12 and the reference positions 13 and 14 are respectively given by: ##EQU1## where b is the base length between the two lenses. From the relationship between equations (1) and (2), the coordinate position of the object 10 to be measured can be expressed by: ##EQU2## More specifically, the coordinate position of the object to be measured can be determined on the basis of the positions of the images of the object to be measured formed on the imaging surface 7. Depending on the applications of measurement results, a predetermined objective is often achieved when only x.sub.0 can be determined. In such a case, x.sub.0 can be determined by observing the shift amount (p.sub.1 -p.sub.2) between the two images before calculating the values p.sub.1 and p.sub.2. In many auto-focus cameras, the defocus amount of a lens need only be determined for focus adjustment, and in this case, focus adjustment can be performed by observing only the shift amount (p.sub.1 -p.sub.2) in place of calculating x.sub.0. Therefore, in this application, the distance and image shift have the same meaning unless otherwise specified.
Normally, an object has an area, and often has a predetermined spread on the imaging surface due to, e.g., the blur of a lens. In such a case, it is difficult to detect the corresponding points of an identical object on the imaging surface. In view of this problem, the following method is normally used. That is, a correlation calculation is performed for two image signals output from the image recording means 8 and 9, and the shift amount is measured on the basis of the calculation result.
Various correlation calculation methods have been disclosed. In the following description, a very simple example will be briefly explained.
FIGS. 9A to 9C are graphs for explaining the concept of the correlation calculation. In an auto-focus camera, in order to convert two images formed by the stereoscopic method into an electrical signal, a line sensor obtained by linearly arranging very small photoelectric conversion light-receiving elements is used. An abscissa k in FIGS. 9A and 9B plots the pixel layout of the line sensor. In FIGS. 9A and 9B, the leftmost pixel is assumed to be the 0th pixel (k=0), and 1st, 2nd,. . ., nth pixels are set in turn to the right in FIGS. 9A and 9B (k=1, 2, . . . , n). The ordinate plots the signal output values of the respective pixels. These signal outputs f.sub.1 (k) and f.sub.2 (k) represent the output values of the above-mentioned two image signals. Note that the pattern of f.sub.2 (k) is the same as that of f.sub.1 (k) but is shifted to the right by d pixels. Equation (5) below is an example of a correlation calculation formula: ##EQU3## FIG. 9C shows an example of the result R(d). In this case, d.sub.min that minimizes R(d) is a value representing the shift between two images (i.e., corresponding to (p.sub.1 -p.sub.2) above).
In order to obtain a correct result by the correlation calculation, the image signals f.sub.1 (k) and f.sub.2 (k) must have a certain contrast, and even when the correlation calculation is performed using flat image signals without any contrast, the shift amount cannot be accurately calculated, and the calculation result includes large errors.
Also, when images of a plurality of objects having different distances are formed on an identical sensor, the distance cannot be accurately calculated, either. In order to prevent a plurality of images from being formed, the widths of the image detection means 8 and 9 shown in FIG. 8 must be reduced, and as a result, the distance measurement range is limited. For example, in the example shown in FIG. 8, when the widths of the image detection means 8 and 9 are reduced to have the reference positions 13 and 14 as the centers, the distance measurement range is limited to an object near the x-axis.
Furthermore, in the method using the line sensor as the image detection means, distance measurements can be performed for only an object near a plane of z=0 in FIG. 8.
In order to simultaneously measure objects present in more directions, a method using a plurality of optical systems and sensors, and a method of performing calculations by dividing the area of a relatively large line sensor into a plurality of areas have been disclosed. However, with these methods, measurements in arbitrary directions cannot be simultaneously performed.
As a means for simultaneously measuring the distances to a plurality of objects present in arbitrary directions, a method using an area sensor obtained by two-dimensionally arranging very small light-receiving elements in place of the line sensor has been disclosed (Japanese Patent Publication No. 4-67607). According to this method, when an image shift detection is performed for a portion in a two-dimensional image signal output from the area sensor, image shifts of objects in arbitrary directions including the z-direction can be simultaneously detected.
When a multi-direction distance measurement is performed using this method, the layout of objects in a space or the shapes of individual objects can be recognized, and hence, this method can be used as an environment recognition means.
Upon executing measurements in the respective directions, a procedure for extracting a portion in a two-dimensional image signal corresponding to the direction of interest as an object for shift amount observation is required. Normally, this extracted signal consists of a rectangular block of n pixels!.times.m pixels! (n, m: integers). FIG. 10 is a view for explaining the block extraction procedure. The coordinate system in FIG. 10 is the same as that shown in FIGS. 9A to 9C. Referring to FIG. 10, 8a and 9a represent area sensors. Lattices shown on the area sensors 8a and 9a represent the pixel layouts of very small light-receiving elements. In order to calculate the distance to an object in a predetermined direction, the output signal from a rectangular block area 15 having n pixels!.times.m pixels! on the sensor 8a is extracted. By changing the position of this block, a measurement in an arbitrary direction can be realized. A signal corresponding to an image signal in a block 16 on the sensor 9a is extracted, and a correlation calculation with the signal on the sensor 8a is performed. In this case, correlation calculations with a plurality of blocks are performed while shifting the extraction position in the image shift direction (y-direction). For example, in FIG. 10, correlation calculations are performed between signals in a plurality of n pixels!.times.m pixels! blocks included in the range of a calculation target area 16 of l pixels!.times.m pixels! (l&gt;n) indicated by a dotted line on the sensor 9a, and the signal in the block 15, thereby calculating the image shift amount. Note that the position and the range of the calculation target area 16 are set on the basis of the state of the optical system used, the measurement purpose, and the nature of the object to be measured.
In the above description, the extraction position of the output signal from only one sensor is shifted. Also, similar calculations may be made by shifting the extraction positions of the two blocks with respect to each other.
Conventionally, in a series of distance measurement operations, the signal extraction block has a rectangular shape, and its size is fixed.
If image shift detection is performed using a relatively small signal extraction block, high measurement precision is assured when the image in the block has a sufficiently high contrast. However, when the block is small, an image with a high contrast is often not included in the block. In such a case, it is difficult to perform shift detection, and large measurement errors occur frequently.
Conversely, when an image shift detection is performed using a large block, an image with a high contrast is included in the block with high probability, and large measurement errors hardly occur unlike in the measurement using the small block. However, when the block is large, the absolute distance measurement precision often lowers as compared to the measurement of a high-contrast image using a small block.
In the case of a large block, images of a plurality of objects having different distances may also be formed, and such images disturb an accurate distance calculation. Such problem is called "perspective conflict".
Owing to the above-mentioned nature, the conventional method suffers from the following problems:
An appropriate distance measurement that can satisfy the required measurement precision cannot be performed.
An appropriate distance measurement cannot be performed in correspondence with the contrast of an image of the object to be measured.
The influence of perspective conflict cannot be properly avoided.
A conventional camera capable of multi-point distance measurement adopts, as a method of selecting a final distance measurement point, a method of selecting the "nearest" distance measurement area from a plurality of distance measurement results. However, since this method often causes distance measurement errors, a selection method having the following algorithm has been proposed (Japanese Laid-Open Patent Application No. 2-282234). That is, in this algorithm, for example, when the distance measurement results at three points have specific patterns, i.e., far, middle, and near patterns, the distance measurement point corresponding to "middle" is selected (since "near" may often correspond to the ground surface or the like).
The method of selecting a distance measurement point when the distance measurement is performed using the area sensor will be described below taking as an example a case wherein a scene shown in FIG. 17 is to be photographed.
The photographing frame is divided into many blocks (each including a set of a plurality of pixels on the area sensor), and distance measurements are performed to obtain distance distribution information (distance map), as shown in FIG. 18. In order to determine the layout of objects in the object field space, objects that constitute the space are grouped.
FIG. 19 shows the grouping result on the basis of the distance map data shown in FIG. 18, and areas are divided in units of objects as a result of the grouping.
As an actual grouping method, some methods are generally known. The simplest example is a method of determining, for all adjacent blocks, that two blocks form an identical object if the difference between the distances (or defocus amounts) of the two adjacent blocks is equal to or smaller than a predetermined value.
With the above-mentioned method, the frame is divided (grouped) into areas in units of objects that constitute the photographing space, as shown in FIG. 19.
Thereafter, the area of a main object is determined from the objects that constitute the photographing space on the basis of, e.g., distance information, size information of the objects, the position information of the objects on the frame, and the like.
However, in the above-mentioned selection method, in a scene shown in FIG. 20, i.e., a scene where a table 2 is present in front of a person 1 as a main object, since the distance between the table 2 and the person 1 is small, an area for measuring the distance to the table 2 and tableware is undesirably selected. Therefore, in this case, the photographer must switch the distance measurement area selection mode to the manual mode so as to select a desired distance measurement area, and must manually select the distance measurement area, resulting in cumbersome operations.